A comprehensive evaluation framework for deep model robustness

Deep neural networks (DNNs) have achieved remarkable performance across a wide range of applications, while they are vulnerable to adversarial examples, which motivates the evaluation and benchmark model robustness. However, current evaluations usually use simple metrics study defenses, far from understanding limitation weaknesses these defense methods. Thus, most proposed defenses quickly shown be attacked successfully, results in “arm race” phenomenon between attack defense. To mitigate this problem, we establish robustness framework containing 23 comprehensive rigorous metrics, consider two key perspectives learning (i.e., data model). Through neuron coverage imperceptibility, data-oriented measure integrity test examples; by delving into structure behavior, exploit model-oriented further evaluate setting. fully demonstrate effectiveness our framework, conduct large-scale experiments on multiple datasets including CIFAR-10, SVHN, ImageNet using different models with open-source platform. Overall, paper provides where researchers could fast toolkit, analytical inspire deeper improvement